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REMARKS/ARGUMENTS 



Applicants acknowledge the Examiner's review of the specification and claims. The 
amendments and remarks presented herein are fully supported by the application as originally 
filed. No new matter has been entered. 

Status of Claims 

Claims 1-15 are pending in the application. 



The present invention concems a system that is capable of inspecting an object — and in 
particular taking a three dimensional measurement of the object - from a single image of the 
object taken by a single camera that is fixed with respect to the object when the image is 
taken. Because there is a single camera taking a sin^e image from a fixed position, the 
captured image cannot properly represent the object in three dimensions. In particular, the 
true or "real world" position of the upper edges of the object will not be correctly represented 
by the raw captured image data. This is because the raw captured image data can only record 
the position of the object in two dimensions (i.e. the X-Y plane defined by the work surface) 
and, in the case of upper edges of the object (which are necessarily offset above the work 
surface, because of the thickness of the object) the offset above the work surface causes the 
X-Y position of the upper edge as recorded in the captured image data to be different from 
the real world X-Y position. This is sometimes referred to as a "parallax error". 

Conventionally, this problem would be overcome by taking multiple images of the object and 
combining the data from the multiple images to produce a real world representation of the 
object. Multiple images are obtained either by moving the camera and object relative to one 
another, by using multiple cameras in different positions relative to the object, and/or by 
using lasers or other structured light patterns to perform a scanning function. However, these 
options are considered to be expensive. 



Overview 
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The present invention avoids the expense of multiple cameras, moving parts, scanning lasers 
or the like, by using a single camera in a fixed position with respect to the object. In order to 
overcome the parallax problem outlined above, the system of the invention processes the 
captured image data, after it has been projected onto the object plane, in the manner described 
in Claim 1 and, in preferred embodiments, as recited in the dependent claims. 

In contrast, US 6,571,196 (Kosuge) discloses an inspection system in which the object (2) to 
be inspected is movable with respect to the camera (1) by means of an XY stage (6) and an 
XY driver (see Figure 2 and column 4, lines 14 to 18). The particular problem being 
addressed by Kosuge is to provide a technique that allows contours of the object to be 
determined in two dimensions in cases where the contour is rounded or the captured image is 
obscure (see Abstract, column 1, lines 55 to 64 and colunm 3, lines 33 to 53). Kosuge is not 
concerned with, and does not disclose, how captured two dimensional image data could be 
converted into three dimensional "real world" data. More specifically, Kosuge describes a 
technology which allows sub-pixel detection of edge position in an obscure 2D image. The 
2D image edge locations are then compared to the 2D image edge locations of a reference 
object. 

With regard to US 5,974,169 (Bachelder), Bachelder describes a technology to determine the 
position, size, centre of mass, boundary and orientation of 2D shapes in a 2D image. 
Bachelder does not detail how real world 3D object measurements are calculated from the 2D 
image shapes. More specifically, Bachelder is concerned with recognising characteristics, 
e.g. an edge of an object, in a 2D image to facilitate the inspection and/or placement of the 
object (see column 1, lines 7 to 21 and column 3, lines 22 to 30). In particular, Bachelder's 
invention is concemed with creating, "bounding boxes" to help identify the object's 
boundaries in the 2D image (see abstract). Bachelder is not concemed with generating, from 
the captured image data, three dimensional data representing the object, although Bachelder 
does compare the image data with a reference model in order to help identify the image 
characteristics. 
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With regard to US 6,604,795 (Buckley), Buckley describes a technology to measure the 3D 
shape of an object by moving the object in front of the camera and by shining a structured 
light, specifically laser light, onto the object. While Buckley's system does measure the real 
world 3D object shape, it requires the use of lasers or other structured light patterns and 
multiple images. Furthermore, it requires that the object be moved by scanning the object 
underneath the sensors (see Figure 1 and columns 5, lines 57 to column 6, line 5), or uses 
multiple cameras for inspecting all sides of the object in a single pass (see Figure 5). In 
either case, Buckley uses triangulation as the means for constructing a model of the object 
(column 5, lines 65 to column 6 line 5) since multiple images are provided. 

In summary, none of Kosuge, Bachelder or Buckley are attempting the same task as is 
addressed by the present invention, namely to produce three dimensional object data from a 
single captured image. Kosuge, Bachelder and Buckley each have movable cameras, 
movable objects, multiple cameras or scanning lasers and so are able to take multiple images 
of the object. This means that they do not need to employ the techniques recited in the claims 
of the present application since they are able to construct three dimensional object data from 
multiple images. It is considered that the teachings of Kosuge, Bachelder or Buckley, either 
individually or in combination, would therefore lead a skilled person away from the subject 
matter of Claims 1 to 13 towards the conventional solutions described in paragraphs [0004] to 
[0007] of the present Patent application. 

Claim Rejections 35 USC s 103 

Applicant agrees with the Examiner that Kosuge does not disclose the following features of 
Claim 1: 

A. The processing apparatus is arranged to project each image edge data component onto 
the object plane to produce a respective object edge data component in the object 
plane. 

B. The processing apparatus is arranged to determine whether each object edge data 
component relates to an edge of the object that lies on the work surface or to an edge 
of the object that is offset above the work surface. 
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C. The apparatus is arranged to receive the image data components from the camera and 
to generate, using said image data components, three dimensional data representing 
the object. 

D. Wherein in order to generate said three dimensional data the apparatus is arranged to 
identify a plurality of said image data components that represent the position of the 
respective edge component of the object in an image plane. 

However, applicant submits that Claim 1 is also novel over Kosuge by virtue of the 
following feature: 

E. The apparatus is arranged to, upon determining that an object edge at a component 
relates to an edge of the object that is offset above the work surface, adjust the value 
of the object edge data component by an amount depending on the ratio of the size of 
the offset in a direction generally perpendicular with the work surface to the 
perpendicular distance of the camera's focal point from the object plane. 

With regard to Feature E, the Examiner contends that this feature is the same as adjusting the 
field of view of the camera. Applicant respectfully disagrees. Adjusting the field of view of 
a camera is something that is done prior to taking an image. In the passage identified by the 
Examiner (Colum 8, lines 62 to 64), Kosuge discloses that the object to be inspected is 
moved by the XY stage (6) until it comes into the field of view of the optical microscope (1). 
This is done to correctly position the object with respect to the camera and therefore takes 
place prior to the image being captured. Once image data has been captured, subsequently 
adjusting the field of view of the camera will have no effect on the previously captured image 
data. In contrast. Feature E of Claim 1 relates to processing the captured image data after it 
has been captured in order to compensate for parallax error as described above. This is 
achieved by adjusting the value of the object edge data component in the manner defined in 
Feature E. Because this adjustment takes place on image data that has already been captured 
(or more specifically on object edge data that has been created from the image data), it cannot 
be affected by adjusting the camera's field of view, nor could this adjustment be replicated by 
adjusting the camera's field of view. Applicant agrees that adjusting the field of view of the 
camera prior to capturing an image will have an effect on the actual data that is captured but 
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this is a separate issue to that addressed by Feature E of Claim 1. To put this another way, in 
order to perform the invention of Claim 1, the operation of Feature E would still be required 
to address the parallax error problem irrespective of any adjustments made to the field of 
view of the camera prior to capturing the image. One way to avoid needing Feature E would 
be to change the field of view, i.e. move the object or camera, in order to capture multiple 
images and then to combine the multiple images using, for example, the triangulation 
technique taught by Buckley. As explained above, this is a conventional technique that the 
invention is avoiding by using only a single image. 

Further, since Kosuge does not disclose Feature A of Claim 1, i.e. it does not disclose that the 
image edge data components are projected onto the object plane to produce respective object 
edged data component, then it cannot disclose Feature E of Claim 1 since Feature E requires 
an adjustment of the object edged data component. Also, since Kosuge does not disclose 
Feature B of Claim 1, then it cannot disclose Feature E either since these two features are 
mutually dependent. 

More generally, the adjustment defined by Feature E of Claim 1 above is specifically 
intended to compensate for parallax error in the captured image data. This is only a concern 
when trying to generate three dimensional "real world" object data from the captured image 
data. This compensation cannot be effected by adjusting the camera's field of view since, for 
any three dimensional object, the parallax error will be present in the captured image data 
irrespective of any adjustments of the camera's field of view. Since Kosuge is not concerned 
with generating three dimensional real world data firom captured two dimensional image data, 
he does not identify or address the parallax problem and so does not disclose or suggest 
Feature E above. 

The Examiner argues that Bachelder discloses Features A and B of Claim 1 as set out above. 
The Applicant respectfully disagrees. With regard to Features A and B, Bachelder is not 
concerned with generating, from the captured two dimensional image data, three dimensional 
"real world" data representing the object. All of the labelling operations and bounding box 
related operations disclosed by Bachelder are performed on the two dimensional captured 
image data in order to allow Bachelder to determine the position, size, centre of mass, 
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boundary and orientation of two dimensional shapes from the two dimensional captured 
image data (see for example the abstract of Bachelder). The purpose of Bachelder is to 
provide an analysis of the two dimensional captured image data that allow it to be compared 
to expected characteristics of the real world object for inspection purposes (see for example 
column 10 lines 40 to 45). More particularly, the passages referred to by the Examiner, 
namely column 2, lines 31 to 34 and column 2, lines 26 to 29, relating to bounding boxes and 
labelling is performed on the two dimensional captured image data in order to help to identify 
two dimensional shapes in the image data (see for example Claim 1 in which it is clear that 
the bounding box activities are performed on the 2D image) . Bachelder is not concerned with 
projecting the image edge data components onto the object plane to produce a respective 
object edge data component (since it is not concerned with converting the 2D image data into 
corresponding 3D real world data) and so is not concerned with determining whether an 
object edge data component (which is a 3D real world data component) relates to an edge of 
the object that lies on the work surface or offset above the work surface. It is noted that 
Bachelder does make a decision on whether or not a point in the 2D image data may 
correspond to a top edge of the corresponding real world object, but it does this by first 
determining if the point lies in a particular bounding box and then making a comparison with 
a model of the real world object (column 8, lines 32 to 34). However, this is not concerned 
with 3D real world data that has been created from the image data. It can be seen in 
particular from Figures 3C to 3E and 5B and 5C that the bounding boxes relate purely to 
allowing analysis of the two dimensional image data. 

Accordingly, since neither Kosuge or Bachelder disclose any of features A to E above, it is 
respectfully submitted that their individual or combined teachings could not lead a skilled 
person to the invention of Claim 1. It is respectfully submitted therefore that Claim 1 is novel 
and non-obvious in the light of the teachings of Kosuge and Bachelder. 

With regard to Buckley, the Applicant agrees that Buckley discloses an apparatus in which 
the captured image data components from the camera are used to generate three dimensional 
data representing the object. However, Buckley employs a system that uses one camera and a 
laser line scanner, or multiple cameras, to produce its object model using triangulation (see 
Figure 1, column 5, lines 57 to column 6 line 5; Figure 5; and column 5, lines 65 to column 6, 
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line 5). This is the prior art solution identified in paragraphs [0004] to [0007] of the present 
Patent application. This is explained in more detail in the passage identified by the Examiner 
at column 2, lines 42 to 65 of Buckley. Crucially, because Buckley has the capacity to create 
multiple images (either by scanning or using multiple cameras) it has no need whatsoever for 
Features B or D of Claim 1 as set out above. Accordingly, Buckley's teaching could not lead 
a skilled person to the invention of Claim 1 since Buckley essentially provides an alternative 
solution to the problem being addressed by Claim 1. Therefore, Buckley's teaching, either 
taken on its own or combined with Kosuge and Bachelder, would lead a skilled person away 
from the invention. 

It is respectfully submitted that the invention of Claim 1 is not obvious in the light of 
Buckley, or in the light of the combined teachings of Kosuge, Bachelder and/or Buckley. 

Regarding Claim 2, this claim defines how the object edge data component is adjusted in 
situations where an edge profile of the object is perpendicular to the object plane, or is 
undercut. As such, it relates to how the captured image data, after it has been projected into 
the object plane, is adjusted to compensate for parallax error. As described above in relation 
to Claim 1, this adjustment is not equivalent to, and cannot be replicated by, moving the 
object with respect to the field of view of the camera, since the adjustment must be performed 
on the data after the image has been captured. The movement of the object described by 
Kosuge at column 8, lines 62 to 64 occurs prior to an image being captured and so is not 
relevant to the issue of how to adjust the captured image data in order to compensate for 
parallax error. Moreover, since, as explained above, neither Kosuge, Bachelder or Buckley 
are concerned with creating three dimensional object data from a single two dimensional 
image, the problem of compensating for parallax error does not arise and so none of these 
documents disclose, or have the need of, the features of Claim 2. It is respectfully submitted 
therefore that the individual or combined teachings of these documents could not lead a 
skilled person to the subject matter of Claim 2. 



Regarding Claim 3, the same comments apply as are made in relation to Claim 2. With 
regard in particular to the Examiner's reference to colimm 4, lines 64 to 66 of Bachelder, as 
described above, Bachelder is concerned with determining two dimensional shapes from two 
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dimensional image data and not with creating three dimensional object data from two 
dimensional image data. 

Regarding Claim 4, the same conmients apply as are made in relation to Claim 3. 

Regarding Claim 5, the feature of this claim defines a preferred technique by which the 
system of the invention determines whether or not an object edge data component relates to 
an edge of the object that is on the work surface or above the work surface, i.e. a lower or 
upper edge of the object being inspected. 

The Examiner contends that this technique is disclosed by Bachelder. The applicant 
respectfully disagrees. Firstly, as explained above, the techniques taught by Bachelder are all 
performed on the two dimensional image data and not on the object data as required by Claim 
5. Secondly, Bachelder is not concerned with creating three dimensional object data from the 
two dimensional captured image data and so is not interested in whether or not an object edge 
data component relates to a lower or upper edge of the object. Therefore, the feature of 
Claim 5 would be of no use to Bachelder. Bachelder does make a decision as to whether or 
not a boundary point of the image data corresponds to a top edge of the object model (see 
column 8, lines 32 to 46). However, this is achieved by determining whether or not a 
boundary point of the 2D image data lies within a particular bounding box. This is quite 
different from the technique proposed in Claim 5 which involves comparing the value of the 
difference of first and second parameter values against threshold value, and is based on object 
edge data components rather than image components. 

The specific passage of Bachelder (column 9, lines 51 to 63 and 48, 50) identified by the 
Examiner relates specifically to fitting lines 155A to 155D (Figure 5E) to two dimensional 
boundary points 150 (Figure 5D) that have been determined as edge points. Boundary points 
that are outside the bounding boxes as shown in Figures SB and 5C are discarded. It is 
emphasised that this is a two dimensional analysis of two dimensional data in order to 
perform line fitting and so to model the object. This has nothing to do with determining 
whether or not the edge of the object is a lower edge or an upper edge. 
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Moreover, Bachelder does not disclose any of the following features of Claim 5: calculating a 
respective first parameter relating to a notional reference line extending from the object 
edged data component, calculating a second parameter relating to a notional line extending 
between the object data component and a reference point in the object plane, and comparing 
the difference between said first parameter and said second parameter against a threshold 
value. 

It is respectfully submitted therefore that the teaching of Bachelder either on its own or in 
combination with Kosuge does not disclose the features of Claim 5 or could not lead a skilled 
person to the features of Claim 5. 

Regarding Claim 6, the features of this claim are dependent on Claim 5. Since Bachelder 
does not disclose the features of Claim 5, it is respectfully submitted that it cannot disclose 
the features of Claim 6 either. 

With regard to the passage of Bachelder identified by the Examiner (columns 8, lines 32 to 
46 in Figure 3E) in relation to Claim 6, this states clearly that Bachelder decides that a 
boundary point in the two dimensional image data corresponds to a top edge of the object 
simply by deciding whether or not it falls into a particular bounding box. This does not 
involve the notional reference lines, first and second parameters, and threshold values 
comparisons of Claim 5, or the more specific definition of the first parameter given in Claim 
6. 

Regarding Claims 7 and 8, the features of these claims relate to preferred implementations of 
the technique described in Claim 5 and so similar comments apply as are made in relation to 
Claims 5 and 6. 

With regard to Bachelder' s steps 42, 44 and 46 identified by the Examiner in coimection with 
Claim 7, Applicant agrees that Bachelder discloses finding points in bounding boxes and 
comparing the orientation of points with expected orientations thereby categorising points by 
correlating with a model or a real world object. Applicant cannot see any connection 
between this methodology and the techniques disclosed in Claims 5, 6 and 7. Bachelder's 
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technique does not involve the calculation and comparison of parameters as is required by 
Claims 5 to 7. Instead it simply decides whether or not a given point is within a given 
bounding box. Bachelder then has to make a comparison with a real world model in order to 
categorise the point. This is not relevant to the subject matter of Claims 5 to 7. 

Claim 8 relates to a preferred implementation of the technique of Claim 5 and so similar 
comments apply as have been made in connection with Claims 5 to 7 and it is respectfully 
submitted that Bachelder does not disclose the feature of Claim 8. With regard to the 
Examiner's specific comments concerning column 10, lines 19 to 36 and steps 36, 42, 44 and 
46 described by Bachelder, Applicant cannot see any coimection between this disclosure and 
the subject matter of Claim 8. 

It may be useful to note that the subject matter of Claims 5 to 8 relate to the illustrations in 
Figures 3 and 5 of the present application from which it can hopefully be seen that the 
techniques defined in Claims 5 to 8 do not relate to determining whether or not points are in 
bounding boxes, or to the comparison of data with models or real world objects. 

Regarding Claims 9 and 10, Buckely's technique referred to by the Examiner on page 12 of 
the examination report and described at column 6, lines 6 to 67 and column 7, lines 1 to 67, 
column 8, lines 1 to 14 of Buckley are concerned with where the laser light (5) from the laser 
(3) intersect with the object (8). By scanning the object with the laser, Buckley is able to 
build up enough data (by taking multiple images) to create a three dimensional data record of 
the object. Intersection of the laser light with the object (8) has got nothing to do with the 
intersection of the line of sight of the camera's focal point to the edge of the object, as recited 
in Claim 9. Moreover, the technique described in Claim 9 or 10 is only necessary when 
trying to produce three dimensional object data from a single two dimensional image taken 
by the camera. Since Buckley does not use a single image of the object (instead it uses a 
series of images capturing the laser lines as they scan across the object), there is no need for 
Buckley to perform the techniques described in Claim 9 or 10. It is respectfully submitted 
therefore that Buckley does not disclose the features of Claim 9 or 10. 
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With regard to Claims 11 to 13, these are of corresponding scope to Claim 1 and so similar 
comments apply. 

In light of the above amendments and remarks, Applicant respectfully requests 
reconsideration of the present application and respectfully solicits a Notice of Allowance of 
all claims. It is respectfully submitted that the application is in order for allowance. 

Should the Examiner have any questions or suggestions, she is invited to contact the 
undersigned at (616) 975-5506 or at CQllins(a)vglb.com . 



Respectfully submitted, 



By: Van Dyke, Gardner, Linn & Burkhart, LLP 



JAN ANTONIS 





Catherine S. Collins 

Registration No. 37 599 

2851 Charlevoix Drive, S.E., Suite 207 

P.O. Box 888695 

Grand Rapids, MI 49588-8695 

(616) 975-5500 
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